Imaging device, imaging method, and program

ABSTRACT

FIG. 3

TECHNICAL FIELD

The present technology relates to an imaging device, an imaging method,and a program, for example, an imaging device, an imaging method, and aprogram for setting a resolution using features of an image.

BACKGROUND ART

Cameras in the related art image subjects by users focusing on thesubjects desired to be imaged or changing a zoom magnification. Inaddition, there are also cameras continuously performing focusing whiletracking subjects even when the subjects move. As related art, thefollowing technology for imaging a moving subject has been proposed.

In an imaging device capable of communicating with a terminal devicecapable of acquiring positional information, it has been proposed thatthe imaging device determine whether or not a terminal device is presentin an imaging range on the basis of positional information of theimaging device and positional information of the terminal device, anddetect a subject on the basis of a feature quantity extracted from acaptured image and a feature quantity for detecting a subject wearingthe terminal device when the terminal device is present in the imagingrange. In addition, it has also been proposed that a focal length beadjusted so as to focus on the detected subject when the subject isdetected (see Patent Literature 1).

CITATION LIST Patent Literature

Patent Literature 1: JP 2013-251796A

DISCLOSURE OF INVENTION Technical Problem

In recent years, the resolution of captured images has improved, and theamount of data to be processed has been increasing. In addition, forexample, the amount of data to be transmitted from an imaging element toa processing unit processing data has also been increasing. With such anincrease in the amount of data, a processing load increases, forexample, when processing for continuously detecting a subject moving athigh speed is performed, which leads to a likelihood that the processingmay not be able to follow.

The present technology is contrived in view of such circumstances, andmakes it possible to appropriately reduce the amount of data.

Solution to Problem

An imaging device according to an aspect of the present technologyincludes: a control unit which changes a resolution of a captured imageon the basis of distance information, corresponding to the capturedimage, regarding a detected distance to a subject included in the image.

An imaging method according to an aspect of the present technologyincludes: a step of changing a resolution of a captured image on thebasis of distance information, corresponding to the captured image,regarding a detected distance to a subject included in the image.

A program according to an aspect of the present technology is a programcausing a computer to execute a process including: a step of changing aresolution of a captured image on the basis of distance information,corresponding to the captured image, regarding a detected distance to asubject included in the image.

In an imaging device, an imaging method, and a program according to anaspect of the present technology, a resolution of a captured image ischanged on the basis of distance information, corresponding to thecaptured image, regarding a detected distance to a subject included inthe image.

Note that the imaging device may be an independent device, or may be aninternal block constituting one device.

In addition, the program can be provided by being transmitted through atransmission medium or being recorded in a recording medium.

Advantageous Effects of Invention

According to an aspect of the present technology, it is possible toappropriately reduce the amount of data.

Note that the advantageous effects described here are not necessarilylimited and may be any of the advantageous effects described in thepresent disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a configuration which is an embodimentof an imaging device to which the present technology is applied.

FIG. 2 is a diagram illustrating configuration examples of an imagingelement and an image processing unit.

FIG. 3 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 4 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 5 is a diagram illustrating a stacked structure.

FIG. 6 is a flowchart illustrating processing of an imaging element andan image processing unit.

FIG. 7 is a diagram illustrating setting of a resolution correspondingto a distance.

FIG. 8 is a diagram illustrating thinning-out processing.

FIG. 9 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 10 is a diagram illustrating setting of a resolution correspondingto the detection of a moving body.

FIG. 11 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 12 is a diagram illustrating setting of a resolution correspondingto the detection of a person.

FIG. 13 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 14 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 15 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 16 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 17 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 18 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 19 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 20 is a diagram illustrating another configuration example of eachof an imaging element and an image processing unit.

FIG. 21 is a diagram illustrating an example of use of an imagingelement.

FIG. 22 is a diagram illustrating a recording medium.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, a mode for implementing the present technology(hereinafter, referred to as an embodiment) will be described.

FIG. 1 is a block diagram illustrating a configuration example of animaging device 10 which is an example of an electronic apparatus towhich the present technology is applied.

<Configuration of Imaging Device>

As illustrated in FIG. 1, the imaging device 10 includes an optical unitincluding a lens group 21 and the like, an imaging element 22, an imageprocessing unit 23 which is a camera signal processing unit, a framememory 24, a display unit 25, a recording unit 26, an operation unit 27,a power supply 28, and the like. In addition, the imaging device isconfigured such that the image processing unit 23, the frame memory 24,the display unit 25, the recording unit 26, the operation unit 27, thepower supply 28, the driving unit 29, and the communication unit 30 areconnected to each other through a bus line 31.

The lens group 21 takes in incident light (image light) from a subjectto form an image on an imaging surface of the imaging element 22. Theimaging element 22 converts the amount of incident light imaged on theimaging surface by the lens group 21 into an electrical signal in unitsof pixels and outputs the electrical signal obtained by the conversionas a pixel signal.

The display unit 25 is constituted by a panel-type display unit such asa liquid crystal display unit or an organic electro luminescence (EL)display unit, and displays a movie or a still image captured by theimaging element 22. The recording unit 26 records the movie or the stillimage captured by the imaging element 22 in a recording medium such as amemory card, a video tape, or a Digital Versatile Disk (DVD).

The operation unit 27 gives an operation instruction for variousfunctions of the imaging device 10 in accordance with a user'soperation. The power supply 28 appropriately supplies various types ofpower serving as operation power for the image processing unit 23, theframe memory 24, the display unit 25, the recording unit 26, theoperation unit 27, and the driving unit 29 to these objects to besupplied.

The driving unit 29 controls the driving of a lens constituting the lensgroup 21 to perform control for focusing (so-called auto-focus). Thecommunication unit 30 transmits and receives data to and from anotherdevice through a network.

Such an imaging device 10 is applied to a video camera, a digital stillcamera, and a camera module for a mobile apparatus such as a smartphoneor a mobile phone.

The imaging device 10 having such a configuration has a function ofchanging a resolution in accordance with features of a captured image aswill be described below. The features of the captured image include, forexample, the distance and the size of a subject as will become apparentin order. In addition, a resolution may be changed such that, forexample, the resolution of a distant subject is increased or theresolution of a near subject is lowered.

Note that a resolution may correspond to the size of an image and can berepresented by, for example, the number of horizontal pixels and thenumber of vertical pixels forming one screen. In addition, a resolutioncan be represented as the number of pixels per unit area with respect topixels included in image data in units of frames constituting videodata.

For example, the number of pixels arranged in a pixel array portion 101(FIG. 2) of the imaging element 22 can be used as a value representing aresolution. In a case in which the number of pixels of the imagingelement 22 is set to be a resolution, the resolution of a captured imagemay be the same in images. However, according to the present technology,images having different resolutions can be adopted as will be describedlater.

For example, in one image, the resolutions of some regions may be set tobe higher or lower than the resolutions of other regions. Althoughdetails will be described later, a resolution at the time of imaging maybe set to be the number of pixels of the imaging element 22, and thenumber of pixels in a predetermined region within a captured image maybe reduced (pixels are thinned), whereby the resolution of the region islowered.

In addition, a resolution at the time of imaging may be set to be thenumber of pixels of the imaging element 22, and the number of pixels ismade to be larger than the number of pixels at the time of imaging byperforming processing (processing referred to as up-conversion or thelike) for increasing the resolution of a predetermined region within acaptured image, whereby the resolution of the region is increased.

In this manner, regarding a resolution in this specification, one imagemay have a plurality of different resolutions, and thus a change of aresolution may be processing including change of a resolution at thetime of imaging performed according to the number of pixels of theimaging element 22. In addition, a change of a resolution in thisspecification means that the number of pixels in a predetermined regionwithin an image is changed, for example, a change of reducing orincreasing the number of pixels may be performed.

Further, in other words, since the amount of data may be reduced due toa change in the number of pixels, for example, a decrease in the numberof pixels, it is thereby possible to reduce the amount of data bylowering a resolution. Accordingly, a change of a resolution also meansa change of the amount of data.

In this manner, configurations of the imaging element 22 and the imageprocessing unit 23 in the imaging device 10 having a function ofchanging a resolution in accordance with features of a captured imagewill be described below. First, an example of a case in which thedistance to a subject is used as a feature of a captured image will bedescribed.

<Configurations of Imaging Element and Image Processing Unit>

FIG. 2 is a diagram illustrating configuration examples of the imagingelement 22 and the image processing unit 23 in a case in which thedistance to a subject is used as a feature of a captured image.

An imaging element 22A illustrated in FIG. 2 includes a pixel arrayportion 101, a read-out control unit 102, a memory interface 103, amemory 104, a signal processing unit 105, an output control unit 106, aresolution control unit 107, a resolution map creating unit 108, and adistance information generation unit 109. An image processing unit 23Aincludes a captured image processing unit 121 that performs processingsuch as demosaic processing on a captured image.

The imaging element 22A includes the pixel array portion 101 in which aplurality of Charge Coupled Devices (CCD), Complementary Metal-OxideSemiconductor (CMOS) elements, or the like are arrangedtwo-dimensionally.

The read-out control unit 102 controls the read-out of a signal fromeach pixel of the pixel array portion 101. The read-out control unit 102is configured to be capable of performing control of sequentiallyreading out signals from all of the pixels constituting the pixel arrayportion 101 and control of reading out (a signal from a predeterminedpixel may not be read out) a signal from a predetermined pixel.

The memory interface 103 performs control of supplying a signal from apixel read out by the read-out control unit 102 to the memory 104 orcontrol of supplying a signal held by the memory 104 to the signalprocessing unit 105.

The signal processing unit 105 executes various processing, such asnoise removal, on a signal. The output control unit 106 performs controlfor outputting a signal having been subjected to processing such asnoise removal to the image processing unit 23A. The output control unit106 may be configured to output one stream or may be configured to becapable of outputting N streams in parallel.

The resolution control unit 107 performs control for realizing a setresolution. As described above, a resolution is changed in accordancewith features of a captured image, and the resolution control unit 107controls each of the read-out control unit 102, the memory interface103, the signal processing unit 105, and the output control unit 106 inorder that the changed resolution is obtained.

Control performed by the resolution control unit 107 varies depending onhow the set resolution is realized. However, for example, in a case inwhich a resolution is lowered, the resolution can be lowered by settingthe number of pixels of the pixel array portion 101 from which a signalis read out to be smaller than that in a normal case, and the resolutioncontrol unit 107 controlling the read-out control unit 102 so that suchread-out is performed.

For example, it is possible to lower a resolution to a half by readingout the number of pixels which is half the number of pixels of the pixelarray portion 101. When such control is performed, so that the number ofsignals (the amount of data) supplied from the read-out control unit 102to the memory interface 103 and the subsequent units is reduced, it isthus possible to reduce a processing load on each unit.

In addition, as will be described later, a resolution may be lowered byreading out signals corresponding to all of the pixels from the pixelarray portion 101 and generating new pixel values by performingaddition, division, or the like on the read-out signals. In this case,for example, signals corresponding to all of the pixels are temporarilyheld in the memory 104 and processing such as addition or division isperformed on the signals to generate new pixel values, and the generatednew pixel values are output to the signal processing unit 105. Accordingto such processing, for example, an image having half the number ofpixels is generated.

In a case in which control is performed in this manner, the number ofsignals supplied from the memory interface 103 to the signal processingunit 105 is reduced, and thus it is possible to reduce a processing loadon the signal processing unit 105 and the subsequent units.

It is also possible to lower a resolution with respect to the entireimage to be generated and to lower a resolution for each ofpredetermined regions (pixels) of the image. Lowering of a resolutionfor each predetermined region of the image means that regions havingdifferent resolutions such as a region having a resolution A, a regionhaving a resolution B, and a region having a resolution C may be presentin one image, and means that a different resolution can be set for eachregion.

Here, a description for each region is given, but it is also possible togive a description for each pixel. A change of a resolution of for eachpixel means that a signal is read out or is not read out from apredetermined pixel. In addition, it is also possible to change aresolution for each image. For example, when a video is captured, it isalso possible to set a different resolution for each frame.

In addition, here, a case in which a resolution is lowered will bedescribed as an example. In such a case, processing is performed with ahigh resolution in a normal case, and a resolution becomes lower thanthe resolution in a normal case due to features of a captured image. Inother words, imaging is performed with a high resolution, and aresolution within a predetermined region in an image having this highresolution is lowered. In this manner, an image having a region with ahigh resolution and a region with a lowered resolution in one image isgenerated.

An example of such a case will be described, but the present technologycan also be applied in a case of an inverse pattern. That is, it is alsopossible to perform processing with a low resolution in a normal caseand to make a resolution become higher than the resolution in a normalcase due to features of a captured image.

In addition, it is also possible to perform processing with a mediumresolution in a normal case and to make a resolution become higher orlower than the resolution in a normal case due to features of a capturedimage.

Description will return to FIG. 2. The resolution map creating unit 108sets a resolution in one image in accordance with distance informationreceived from the distance information generation unit 109. That is, aregi on in which a resolution is lowered in one image is set. Since thesetting is performed in accordance with a distance to a subject, thedistance information generation unit 109 generates distance informationof the subject and supplies the generated distance information to theresolution map creating unit 108.

The distance information generation unit 109 generates distanceinformation using information received from the distance informationacquisition unit 131. In the distance information acquisition unit 131and the distance information generation unit 109, a distance to asubject is measured, but the measurement can be performed by a distancemeasurement sensor using, for example, active light (infrared light orthe like). For the distance measurement sensor using active light, aTime-of-Flight (TOF) system, a Structured Light system, or the like canbe applied.

In addition, a stereo camera may be used as the distance informationacquisition unit 131, and the distance information generation unit 109may generate distance information using an image captured by the stereocamera. In addition, a multi-camera may be used as the distanceinformation acquisition unit 131, and distance information may begenerated using a method based on a three-dimensional reconstructionusing the multi-camera.

In addition, the distance information acquisition unit 131 may beconfigured to acquire distance information using an ultrasound sensor.In addition, the distance information acquisition unit 131 may beconfigured to acquire distance information by a method using amillimeter wave radar. In addition, a method using a light field cameracan also be applied.

In addition, the distance information acquisition unit 131 may be set tobe a pixel for phase difference detection, and the distance informationgeneration unit 109 may generate distance information using a signalfrom the pixel for phase difference detection. In a case in whichdistance information is generated using the signal from the pixel forphase difference detection, the pixel for phase difference detection canbe provided in the pixel array portion 101. In addition, focus detectionprocessing may also be performed using the signal from the pixel forphase difference detection.

In this manner, distance information is acquired and generated by thedistance information acquisition unit 131 and the distance informationgeneration unit 109. Depending on a configuration (by which methoddistance measurement is performed), the distance information acquisitionunit 131 may be configured to be included in the imaging element 22A ormay be provided separately from the imaging element 22A.

FIG. 3 illustrates another configuration example of each of the imagingelement 22 and the image processing unit 23. In the configurationillustrated in FIG. 3, the same components as those illustrated in FIG.2 are denoted by the same reference numerals and signs, and adescription thereof will be appropriately omitted.

Comparing configurations of an imaging element 22B and an imageprocessing unit 23B illustrated in FIG. 3 with configurations of theimaging element 22A and the image processing unit 23A illustrated inFIG. 2, the internal configurations thereof are different from eachother. The imaging element 22B illustrated in FIG. 3 is configured toinclude a pixel array portion 101, a read-out control unit 102, a memoryinterface 103, a memory 104, a signal processing unit 105, an outputcontrol unit 106, and a resolution control unit 107.

The image processing unit 23B illustrated in FIG. 3 is configured toinclude a resolution map creating unit 108, a distance informationgeneration unit 109, and a captured image processing unit 121. Theconfiguration illustrated in FIG. 3 is different from the configurationillustrated in FIG. 2 in that the resolution map creating unit 108 andthe distance information generation unit 109 included in the imagingelement 22A are included in the image processing unit 23B.

In this manner, functions constituting the imaging element 22 and theimage processing unit 23 can be allocated to the imaging element 22 orthe image processing unit 23.

The configurations of the imaging element 22A and the image processingunit 23A illustrated in FIG. 2 or the configurations of the imagingelement 22B and the image processing unit 23B illustrated in FIG. 3 areconfigurations in which a resolution is changed on the imaging element22A side. In a case in which such configurations are adopted, the amountof data to be supplied from the imaging element 22A to the imageprocessing unit 23A is reduced according to a degree by which aresolution is set to be lowered when a low resolution is set.

Next, a case in which a configuration in which a resolution is changedon the image processing unit 23 side will be described. In a case inwhich such a configuration is adopted, the amount of data to be outputfrom the image processing unit 23 is reduced according to a degree bywhich a resolution is set to be lowered when a low resolution is set.Therefore, it is possible to reduce the amount of data flowing through anetwork, for example, when the data is supplied to another devicethrough the network, or the like.

FIG. 4 is a diagram illustrating configuration examples of the imagingelement 22 and the image processing unit 23 in a case in which aresolution is changed on the image processing unit 23 side. In theconfiguration illustrated in FIG. 4, the same components as thoseillustrated in FIG. 2 are denoted by the same reference numerals andsigns, and a description thereof will be appropriately omitted.

An imaging element 22C illustrated in FIG. 4 is configured to include apixel array portion 101, a read-out control unit 102, a signalprocessing unit 105, and an output control unit 106. The imaging element22C illustrated in FIG. 4 does not have a function of changing aresolution, and is thus configured such that, for example, the memoryinterface 103, the memory 104, and the resolution control unit 107 areomitted from the imaging element 22A illustrated in FIG. 2.

Note that, here, although description will continue on the assumptionthat a memory 14 is provided to temporarily hold signals in order toperform a process of changing a resolution, a configuration may be, ofcourse, adopted in which a memory interface 103 and a memory 14 areincluded in a case in which a mechanism for temporarily holding signalsis also desired to be provided for processes other than the process ofchanging a resolution, or the like.

The image processing unit 23C illustrated in FIG. 4 is configured toinclude a resolution control unit 107, a distance information generationunit 109, and a captured image processing unit 121. Distance informationgenerated by the distance information generation unit 109 is supplied tothe resolution control unit 107.

The resolution control unit 107 thins out pixels in an image to beprocessed by the captured image processing unit 121 in accordance with aresolution and outputs a result of the thinning-out to a processing unitin a latter stage, for example, a communication unit (not shown)communicating with another device, or the like through the network.

In a case of such configuration, for example, it is possible to outputimage data processed by the image processing unit 23C to the networkthrough the communication unit 30 and to reduce the amount of data whenthe data is output to another device through the network.

The imaging element 22 and the image processing unit 23 illustrated inFIGS. 2 to 4 can be constituted by a stacked image sensor in which aplurality of substrates (dies) are stacked. Here, a description will begiven of a case in which the imaging element 22 and the image processingunit 23 are constituted by a stacked image sensor using the imagingelement 22 and the image processing unit 23 illustrated in FIG. 2 or 3as examples.

FIG. 5 is a diagram illustrating a configuration example of a stackedimage sensor in which all of the imaging element 22 and the imageprocessing unit 23 of FIG. 3 are built in. The stacked image sensor ofFIG. 5 has a two-layer structure in which a pixel substrate 201 and asignal processing substrate 202 are stacked on each other.

The pixel array portion 101, the read-out control unit 102, the memoryinterface 103, the memory 104, the signal processing unit 105, and theoutput control unit 106 included in the imaging element 22 are formed onthe pixel substrate 201. In addition, the distance informationacquisition unit 131 is also formed on the pixel substrate 201.

In a case in which distance information is obtained using, for example,a TOF system, the distance information acquisition unit 131 isconfigured to include an irradiation unit irradiating a subject withpredetermined light and an imaging element receiving the irradiationlight. It is possible to form portions on the pixel substrate 201inclusive of a portion of the imaging element constituting the distanceinformation acquisition unit 131 or a portion such as the irradiationunit. Further, also in a case in which the distance informationacquisition unit 131 is formed by a pixel for phase differencedetection, and the like, the distance information acquisition unit 131is formed on the pixel substrate 201.

The resolution control unit 107, the resolution map creating unit 108,the distance information generation unit 109, and the captured imageprocessing unit 121 are formed on the signal processing substrate 202.

In this manner, the imaging element 22 and the image processing unit 23may be formed as a stacked image sensor. Note that a change can beappropriately made such that only the imaging element 22 is configuredas a stacked image sensor.

In addition, the stacked image sensor can be configured as a stackedsubstrate of four or more layers in addition to a two-layer substrate ora three-layer substrate. For example, the memory 104 is provided on alayer (memory layer) other than the pixel substrate 201 and the signalprocessing substrate 202, and the memory layer may be configured to beprovided between the pixel substrate 201 and the signal processingsubstrate 202 or below the signal processing substrate 202.

<Change of Resolution Based on Distance>

Reference will be made to a flowchart of FIG. 6 to describe a case inwhich a resolution is changed on the basis of a distance in theconfigurations of the imaging element 22 and the image processing unit23 illustrated in FIGS. 2 to 4.

In step S101, distance information is generated. The distanceinformation generation unit 109 (for example, FIG. 3) generates distanceinformation on the basis of information to be acquired by the distanceinformation acquisition unit 131 (FIG. 3). The distance information canbe generated by applying a TOF system or the like as described above.

In step S102, a resolution map is created. The resolution map creatingunit 108 creates the resolution map on the basis of the distanceinformation generated by the distance information generation unit 109.The resolution map is a map useful as information in which a region(pixel) in which a resolution is not desired to be lowered and a region(pixel) which is not useful as information and in which a resolution isdesired to be lowered in an image are written.

Further, in a case in which a resolution is set in units of frames, aresolution map is set to be a map in which a resolution in a frame(image) is set to be uniformly high in a case of the frame which isuseful as information and in which a resolution is not desired to belowered, and is set to be a map in which a resolution in a frame (image)is set to be uniformly low in a case of the frame which is not useful asinformation and in which a resolution is lowered.

The distance information generation unit 109 generates, for example, adepth map (depth map) as distance information, and the resolution mapcreating unit 108 may generate a resolution map by detecting a regionhaving a predetermined threshold value or less and a region having thepredetermined threshold value or greater in the generated depth map. Aconfiguration may also be adopted in which a plurality of resolutionsare set by providing a plurality of threshold values.

In step S103, a thinning-out rate is adjusted on the basis of theresolution map to generate a captured image (output data). Thethinning-out rate is a rate which is set to achieve a set resolution,and has a value as described with reference to FIG. 7.

A description of such processing will be further added with reference toFIG. 7. A of FIG. 7 illustrates an example of a captured image (set tobe an input image 301). In the input image 301, four persons are imagedon the upper left side, and one person is imaged in the front. Further,in the input image 301, a tree is imaged on the upper right side, and acar is imaged at the center.

In a case in which, for example, a depth map is generated from distanceinformation when such input image 301 is acquired, a distance image 302as illustrated in B of FIG. 7 is obtained. In B of FIG. 7, a darkercolor (close to black) means a longer distance, and a lighter color(close to white) means a shorter distance.

Note that the distance is a distance from the imaging device 10, and forexample, the short distance means that a distance from the imagingdevice 10 to a subject is short. Alternatively, the distance may be adistance in a relative positional relationship between subjects in animage. It may be determined that the distance is long in a case in whicha predetermined subject is set as a reference and another is separatedfrom the reference subject, and it may be determined that the distanceis short in a case in which another subject is close to the referencesubject. Nearness and distantness can be relatively determined using thepredetermined subject as a reference.

In addition, the distance may be absolute distance information obtainedin accordance with detection performed through processing of thedistance information acquisition unit 131 and the distance informationgeneration unit 109. The distance information may be, for example, animaging distance. For example, the imaging distance is obtained using 10meters as a reference, and it can be determined that a distance is longfor a subject positioned farther than 10 meters and a distance is shortfor a subject positioned closer than 10 meters.

The reference imaging distance is not limited to 10 meters, and may beset to any value or may vary depending on an imaging scene. For example,in a case in which a distant view is imaged like a near view by zoomingor the like, it may be determined that a distance is short for theentire image, and processing may be performed with a resolution obtainedwhen a distance is short.

In a case in which the distance image 302 as illustrated in B of FIG. 7is obtained as distance information, a resolution is set for each pixel(a region having a predetermined size) of the distance image 302. C ofFIG. 7 illustrates an example in which a resolution is set for eachregion having a predetermined size, and is a diagram illustrating anexample of a resolution map.

C of FIG. 7 illustrates that the smaller a numerical value is, the lowera thinning-out rate is, and the larger a numerical value is, the highera thinning-out rate is. In other words, C of FIG. 7 illustrates that thesmaller a numerical value is, the higher a resolution is, and the largera numerical value is, the lower a resolution is.

Referring to a resolution map 303 illustrated in C of FIG. 7, athinning-out rate in a near region (near view) is set to be high (largeas a numerical value, and a resolution is set to be low), and athinning-out rate in a distant region (distant view) is set to be low(small as a numerical value, and a resolution is set to be high).

There is a strong likelihood that a subject in a distant view may beimaged in a small size, and there is a likelihood that the subject maynot be visually recognized when the region of the subject in a distantview is thinned out (resolution is lowered). Accordingly, the resolutionof the subject in a distant view is maintained or increased (athinning-out rate is set to be low).

In addition, there is a strong likelihood that the subject in a distantview may be imaged in a small size, and there is a likelihood that thesubject having a small size may be changed to a larger size so as to bevisually recognized. Such a region which is likely to be enlargedlydisplayed after being imaged is imaged with a high resolution, and isviewed so as to be enlargedly displayed at a later point in time.Accordingly, the resolution of the subject in a distant view ismaintained or increased (a thinning-out rate is set to be low).

On the other hand, there is a strong likelihood that a subject in a nearview may be imaged in a large size, and there is a slim likelihood thatthe subject may not be visually recognized even when the region of thesubject in a near view is thinned out (resolution is lowered).Accordingly, the resolution of the subject in a near view is lowered (athinning-out rate is set to be high).

Alternatively, in reverse, the resolution of the subject in a near viewmay be increased, and the resolution of the subject in a distant viewmay be lowered. For example, in a case where only the subject in a nearview is set to be an object to be processed, it is possible to performchange such that the resolution of the subject in a near view isincreased and the resolution of the subject in a distant view islowered.

In this manner, how to set a resolution can be appropriately changed inaccordance with the purpose of changing a resolution.

Referring to the resolution map 303 illustrated in C of FIG. 7, aplurality of numerical values are set in the resolution map 303. Thatis, a resolution (thinning-out rate) is set for each of a plurality ofregions with respect to one image. In this case, a resolution(thinning-out rate) is set in accordance with a distance to a subject.

In the resolution map 303 illustrated in C of FIG. 7, numerical valuesof 1 to 8 are set. In this case, a case in which resolutions of eightlevels are set is shown. The number of levels may be arbitrary, and aplurality of resolutions (a plurality of thinning-out rates) can be set.

For example, in a case in which 100 pixels of 10×10 are set to be anobject to be processed, all of 100 pixels within a region having anumerical value of “1” set therein are read out and a resolution ismaintained, for example, within the resolution map 303 illustrated in Cof FIG. 7. In addition, for example, within the resolution map 303illustrated in C of FIG. 7, 80% of pixels (80 pixels) are thinned outfrom a region having a numerical value of “8” set therein, 20 pixelspresent at a predetermined position are read out from 100 pixels, and aresolution is lowered.

In this manner, a resolution map is created from distance information,and a resolution is controlled on the basis of the resolution map.

In the imaging element 22 and the image processing unit 23 illustratedin FIG. 2 or 3, in a case in which the resolution control unit 107controls a resolution on the basis of a resolution map, the resolutioncontrol unit 107 gives an instruction to, for example, the read-outcontrol unit 102 and controls the number of pixels to be read out fromthe pixel array portion 101 to control a resolution.

For example, as in the above-described example, 20 pixels present at apredetermined position are set to be pixels to be read out from 100pixels within a region having a numerical value of “8” set thereinwithin the resolution map 303 illustrated in C of FIG. 7, and pixelvalues are read out from the set pixels. In this case, a resolution iscontrolled by controlling reading-out performed by the read-out controlunit 102.

Alternatively, the resolution control unit 107 gives an instruction tothe memory interface 103 and controls data output from the memoryinterface 103 to the signal processing unit 105 to control a resolution.

For example, as in the above-described example, 20 pixels present at apredetermined position are set to be pixels to be output to the signalprocessing unit 105 from 100 pixels within a region having a numericalvalue of “8” set therein within the resolution map 303 illustrated in Cof FIG. 7, and pixel values read out from the set pixels are read outfrom the memory 104 and output to the signal processing unit 105. Inthis case, a resolution is controlled by controlling reading-outperformed by the memory interface 103.

Alternatively, the resolution control unit 107 may give an instructionto the memory interface 103 and the memory 104 and control data outputfrom the memory interface 103 to the signal processing unit 105 tocontrol a resolution.

For example, as in the above-described example, in a case in which 20pixels present at a predetermined position are set to be pixels to beoutput to the signal processing unit 105 from 100 pixels within a regionhaving a numerical value of “8” set therein within the resolution map303 illustrated in C of FIG. 7, pixel values corresponding to 20 pixelsare generated from pixel values corresponding to 100 pixels held in thememory 104 (the generation will be described later with reference toFIG. 8).

The generated pixel values corresponding to 20 pixels are read out fromthe memory 104 and output to the signal processing unit 105. In thiscase, a resolution is controlled by controlling the memory interface 103and the memory 104.

Such thinning-out processing such as the generation of pixel values of20 pixels from pixel values of 100 pixels can also be performed by thesignal processing unit 105. In a case in which thinning-out processingis performed by the signal processing unit 105, the resolution controlunit 107 gives an instruction to the signal processing unit 105, and thesignal processing unit 105 performs thinning-out processing (processingincluding addition, division, and the like of pixel values), and data tobe output is controlled, whereby a resolution is controlled.

Alternatively, the resolution control unit 107 may give an instructionto the output control unit 106 and may control data to be output fromthe output control unit 106 to the image processing unit 23 to control aresolution.

For example, as in the above-described example, in a case in which 20pixels present at a predetermined position are set to be pixels to beoutput to the image processing unit 23 from 100 pixels within a regionhaving a numerical value of “8” set therein within the resolution map303 illustrated in C of FIG. 7, pixel values of 100 pixels are suppliedto the output control unit 106 from the signal processing unit 105, andthe output control unit 106 outputs pixel values corresponding to 20pixels to the image processing unit 23 from the supplied pixel values of100 pixels. In this case, a resolution is controlled by controlling theoutput control unit 106.

In the imaging element 22C and the image processing unit 23C illustratedin FIG. 4, in a case in which a resolution is controlled on the basis ofa resolution map, the resolution control unit 107 gives an instructionto the captured image processing unit 121 and controls the number ofpixels to be output from the captured image processing unit 121 to thebus line 31, so that a resolution can be controlled.

For example, as in the above-described example, in a case in which pixelvalues corresponding to 20 pixels are output from 100 pixels within aregion having a numerical value of “8” set therein within the resolutionmap 303 illustrated in C of FIG. 7, pixel values corresponding to 100pixels are supplied to (the captured image processing unit 121 within)the image processing unit 23 from the imaging element 22C. The capturedimage processing unit 121 thins out 80 pixels from the supplied 100pixels on the basis of the instruction given from the resolution controlunit 107 and outputs the pixel values corresponding to 20 pixels. Inthis manner, a resolution may be controlled by controlling the capturedimage processing unit 121.

A case in which pixel values are temporarily held in the memory 104 andpixels are thinned out will be described with reference to FIG. 8.

As illustrated in A of FIG. 8, a case in which eight horizontal pixelsand two vertical pixels are processed will be described as an example.In addition, here, a description will continue using a case in which acolor arrangement (color filter) is a Bayer array of RGB (Red, Green,Blue) as an example, but the present technology does not describe thatan application range is limited to such color arrangement.

For example, 2×2 pixels in the horizontal and vertical directions areset to be a repetitive unit, and the present technology can be appliedalso in a case in which red (R)/green (G)/blue (B)/transparent (C) colorfilters are arranged or a case in which red (R)/transparent(C)/transparent (C)/transparent (C) color filters are arranged in therepetitive unit.

In a case of a region in which a resolution is maintained andthinning-out is not performed, in other words, a region in which thereading-out of all pixels is performed, all pixels are read out asillustrated in A of FIG. 8. That is, in this case, pixel values are readout from pixels 401-1 to 401-8 and pixels 402-1 to 402-8.

In a case of the reading-out of all pixels, pixel values of the pixels401-1 to 401-8 and the pixels 402-1 to 402-8 are read out from the pixelarray portion 101 and temporarily held in the memory 104. Thereafter,the pixel values of the pixels 401-1 to 401-8 and the pixels 402-1 to402-8 are read out from the memory 104 and supplied to the signalprocessing unit 105 under the control of the memory interface 103.

In a case of a region in which a resolution is lowered, for example, aregion in which thinning-out to half is performed, addition and divisionare performed, for example, as illustrated in B of FIG. 8, and thuspixel values corresponding to 2 pixels are converted into a pixel valuecorresponding to one pixel. In the example illustrated in B of FIG. 8,for example, pixel values of an R pixel 401-1 and an R pixel 401-3 areadded up and divided by 2, and thus an obtained value is converted intoa pixel value of an R pixel corresponding to one pixel.

That is, a pixel value is calculated by calculating an average value ofpixel values of the adjacent same colors. Similarly, regarding G pixelsand B pixels other than the R pixels, a pixel value is calculated byobtaining an average value.

Note that, here, although thinning-out in the horizontal direction isdescribed, thinning-out in the vertical direction can be performed inthe same manner as the thinning-out in the horizontal direction. Here,thinning-out in the horizontal direction is described, and thinning-outin the vertical direction is performed in the same manner as thethinning-out in the horizontal direction and will not described.

In a case of a region in which a resolution is lowered, for example, aregion in which thinning-out to half is performed, addition and divisionare performed, for example, as illustrated in C of FIG. 8, and thuspixel values corresponding to 2 pixels may be converted into a pixelvalue corresponding to one pixel. In the example illustrated in C ofFIG. 8, for example, pixel values of an R pixel 401-1 and an R pixel401-3 are added up at a ratio of 3:1 and divided by 4, and thus anobtained value is converted into a pixel value of an R pixelcorresponding to one pixel.

That is, in this case, weighted addition is performed, and the weightedaddition value is divided to generate a pixel value corresponding to onepixel. In a case of an R pixel, the proportion of an R pixel (forexample, the R pixel 401-1) positioned on the left side, among adjacentR pixels, is set to 3 the proportion of an R pixel (for example, the Rpixel 401-3) positioned on the right side is set to 1, and additionthereof is performed.

2×2 pixels, which are G pixels, in the horizontal and verticaldirections are set to be a repetitive unit. In a case of a Gr pixelpositioned on the upper right side in the repetitive unit, theproportion of a Gr pixel (for example, a Gr pixel 401-2) positioned onthe left side, among adjacent Gr pixels, is set to 1 and the proportionof a Gr pixel (for example, a Gr pixel 401-4) positioned on the rightside is set to 1, and addition thereof is performed.

2×2 pixels, which are G pixels, in the horizontal and verticaldirections are set to be a repetitive unit. In a case of a Gb pixelpositioned on the lower left side in the repetitive unit, the proportionof a Gb pixel (for example, a Gb pixel 402-1) positioned on the leftside, among adjacent Gb pixels, is set to 3 and the proportion of a Gbpixel (for example, a Gb pixel 402-3) positioned on the right side isset to 1, and addition thereof is performed.

In a case of a B pixel, the proportion of a B pixel (for example, a Bpixel 402-2) positioned on the left side, among adjacent B pixels, isset to 1 and the proportion of a B pixel (for example, a B pixel 402-4)positioned on the right side is set to 1, and addition thereof isperformed.

In this manner, weighting is performed at the time of the addition. Inthis case, weighting of 3:1 or 1:3 may be performed. Naturally, anotherweighting may be performed.

In a case of a region in which a resolution is lowered, for example, aregion in which thinning-out to one third performed, addition anddivision are performed, for example, as illustrated in D of FIG. 8, andthus pixel values corresponding to three pixels are converted into apixel value corresponding to one pixel. In the example illustrated in Dof FIG. 8, for example, pixel values of three pixels of an R pixel401-1, an R pixel 401-3, and an R pixel 401-5 are added up and dividedby 3, and thus an obtained value is converted into a pixel value of an Rpixel corresponding to one pixel.

That is, a pixel value corresponding to one pixel is calculated bycalculating an average value of pixel values of the adjacent threepixels of the same color. Similarly, regarding G pixels and B pixelsother than the R pixels, a pixel value is calculated by obtaining anaverage value.

Here, a case of a thinning-out rate for reducing the number of pixels tohalf and one third has been described as an example. However, similarly,other thinning-out rates can be calculated by obtaining an average valueor calculated by performing weighted addition and division.

Note that an average value is an example, weighting may be performed asdescribed above, and thinning-out processing is not limited to theabove-described method. Thinning-out processing may be performed byother methods.

<Change of Resolution Based on Detection of Moving Body>

In the above-described embodiment, a case in which a resolution ischanged on the basis of a distance has been described. A configurationmay also be adopted in which a resolution is changed on the basis ofinformation other than a distance. Hereinafter, a case in which aresolution is changed on the basis of information other than a distancewill be described. First, a case in which a moving body is detected anda resolution is changed on the basis of the detected moving body will bedescribed.

FIG. 9 is a diagram illustrating configuration examples of the imagingelement 22 and the image processing unit 23 when adopting aconfiguration in which a resolution is changed on the basis of detectionof a moving body. In a configuration of each of an imaging element 22Dand an image processing unit 23D illustrated in FIG. 9 and aconfiguration of each of the imaging element 22B and the imageprocessing unit 23B illustrated in FIG. 3, the same components aredenoted by the same reference numerals and signs, and a descriptionthereof will be appropriately omitted.

Note that although a description will be given of an example ofconfigurations of the imaging element 22 and the image processing unit23 in a case in which a resolution is changed on the basis ofinformation other than a distance to be described below areconfigurations based on the configurations of the imaging element 22Band the image processing unit 23B illustrated in FIG. 3, theconfigurations may be configurations based on the configurations of theimaging element 22A and the image processing unit 23A illustrated inFIG. 2 or may be configurations based on the configurations of theimaging element 22C and the image processing unit 23C illustrated inFIG. 4.

The imaging element 22D illustrated in FIG. 9 has the same configurationas that of the imaging element 22B illustrated in FIG. 3. The imageprocessing unit 23D illustrated in FIG. 9 is different from the imageprocessing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a moving body detection unit 431.

The moving body detection unit 431 of the image processing unit 23Dillustrated in FIG. 9 detects a moving body using a captured imageprocessed by the captured image processing unit 121. The resolution mapcreating unit 108D creates a resolution map on the basis of the movingbody (the region of the moving body) which is detected by the movingbody detection unit 431.

Note that, here, although a description will continue on the assumptionthat the moving body detection unit 431 detects a moving body using acaptured image processed by the captured image processing unit 121, aconfiguration can also be adopted in which the moving body detectionunit 431 detects a moving body using a captured image to be directlysupplied from the imaging element 22D. That is, a configuration can alsobe adopted in which the moving body detection unit 431 receives thesupply of a captured image from a unit other than the captured imageprocessing unit 121.

Also in the following description, a description is given on theassumption that a portion equivalent to the moving body detection unit431 is processed using a captured image processed by the captured imageprocessing unit 121, but a configuration can also be adopted in whichthe supply of a captured image is received from a unit other than thecaptured image processing unit 121.

Processing performed by the imaging element 22D and the image processingunit 23D configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting a moving body.

A description of a case in which a resolution is changed by thedetection of a moving object will be added with reference to FIG. 10. Aof FIG. 10 is a diagram illustrating an example of a captured image (animage to be processed). An input image 301 illustrated in A of FIG. 10is the same as the input image 301 illustrated in A of FIG. 7.

In a case in which such an input image 301 is acquired and a moving bodyis detected using the input image 301, an image 332 which is a result ofmovement detection as illustrated in B of FIG. 10 is obtained. In B ofFIG. 10, a region surrounded by a quadrangle is a region detected as amoving body.

Any method may be used as a method of detecting a moving body. Forexample, a moving body may be detected by comparing pixel values witheach other using the input image 301 for two frames. In addition, asubject within the input image 301 and an image registered as a movingbody in advance, for example, an image of a car are matched to eachother, and the region of the subject may be detected as the region ofthe moving body in a case in which the degree of matching is high.

In a case in which a moving object detection result image 332 asillustrated in B of FIG. 10 is obtained as moving object detectioninformation, a resolution is set for each region of the movementdetection result image 332. C of FIG. 10 is a diagram illustrating anexample in which a resolution is set for each region and illustrating anexample of a resolution map.

A resolution map 333 illustrated in C of FIG. 10 is the same as theresolution map 303 illustrated in C of FIG. 7, and indicates that thesmaller a numerical value within the resolution map 333 is, the lower athinning-out rate is, and the higher a numerical value is, the higher athinning-out rate is. In other words, in C of FIG. 10, the smaller anumerical value is, the higher a resolution is, and the higher anumerical value is, and the lower a resolution is.

A resolution is set to be high (maintained) for a region in which amoving body is detected, and a resolution is set to be low for the otherregions. Basically, such setting is performed, but it is also possibleto set a resolution in combination with a distance to a subject. C ofFIG. 10 illustrates an example of the resolution map 333 in a case inwhich a resolution is set in consideration of such a distance.

For example, a resolution of a moving body in a distant view is set tobe high, and a resolution of a moving body in a near view is set to below. In the example illustrated in C of FIG. 10, a person detected in anupper left region as a moving body is a moving body and is far way, andthus a thinning-out rate is set to a small value such as 1 or 2.Similarly, a person detected in a lower right region is nearby, and thusa thinning-out rate is set to 4 which is a larger value than that of theperson who is far away.

In addition, a high rate of 10 is set as a thinning-out rate for aregion in which a moving body is not detected. For example, in a case inwhich thinning-out rates of 10 levels of 1 to 10 are set, it is possibleto prevent pixel values from being read out from the region when athinning-out rate is 10.

In a case in which it is intended to detect a moving body and performany processing on the moving body, a thinning-out rate is set to 10 fora portion other than the moving body so as to prevent the portion frombeing read out. In this manner, it is possible to drastically reduce theamount of data to be processed.

In this manner, it is also possible to detect a moving body and to set aresolution in accordance with the detected moving body. In addition, itis also possible to set a resolution in accordance with a result ofmovement detection and a distance to the moving body.

Note that, in a case in which a resolution is set using not only aresult of movement detection but also a distance, a configuration can beadopted in which the above-described processing is executed by addingthe distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.9.

<Change of Resolution Based on Detection of Person>

Next, a case in which a person is detected and a resolution is changedon the basis of (the region of) the detected person will be described asan example of a case in which a resolution is changed on the basis ofinformation other than a distance.

FIG. 11 is a diagram illustrating configuration examples of an imagingelement 22E and an image processing unit 23E when adopting aconfiguration in which a resolution is changed on the basis of thedetection of a person. In a configuration of each of the imaging element22E and the image processing unit 23E illustrated in FIG. 11 and aconfiguration of each of the imaging element 22B and the imageprocessing unit 23B illustrated in FIG. 3, the same components aredenoted by the same reference numerals and signs, and a descriptionthereof will be appropriately omitted.

The imaging element 22E illustrated in FIG. 11 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23E illustrated in FIG. 11 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a person detection unit 451.

The person detection unit 451 of the image processing unit 23Eillustrated in FIG. 11 detects a person using a captured image processedby the captured image processing unit 121. A resolution map creatingunit 108E creates a resolution map on the basis of (the region of) theperson detected by the person detection unit 451.

Processing performed by the imaging element 22E and the image processingunit 23E configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting a person.

A description of a case in which a resolution is changed by thedetection of a person will be added with reference to FIG. 12. A of FIG.12 is a diagram illustrating an example of a captured image (an image tobe processed). An input image 301 illustrated in A of FIG. 12 is thesame as the input image 301 illustrated in A of FIG. 7.

In a case in which such an input image 301 is acquired and a person isdetected using the input image 301, an image 352 which is a result ofperson detection as illustrated in B of FIG. 12 is obtained. In B ofFIG. 12, a region surrounded by a quadrangle is a region detected as aperson.

Any method may be used as a method of detecting a person. A subjectwithin the input image 301 and an image registered as a person inadvance are matched to each other, and the region of the subject may bedetected as the region of the person in a case in which the degree ofmatching is high.

In a case in which a person detection result image 352 as illustrated inB of

FIG. 12 is obtained as person detection information, a resolution is setfor each region of the person detection result image 352. C of FIG. 12is a diagram illustrating an example in which a resolution is set foreach region and illustrating an example of a resolution map.

A resolution map 353 illustrated in C of FIG. 12 is the same as theresolution map 303 illustrated in C of FIG. 7, and indicates that thesmaller a numerical value within the resolution map 353 is, the lower athinning-out rate is, and the higher a numerical value is, the higher athinning-out rate is. In other words, in C of FIG. 12, the smaller anumerical value is, the higher a resolution is, and the higher anumerical value is, and the lower a resolution is.

A resolution is set to be high (maintained) for a region in which aperson is detected, and a resolution is set to be low for the otherregions. Basically, such setting is performed, but it is also possibleto set a resolution in combination with a distance to a subject. C ofFIG. 12 illustrates the resolution map 353 in a case in which aresolution is set in consideration of such a distance.

For example, a resolution of a person in a distant view is set to behigh, and a resolution of a person in a near view is set to be low. Inthe example illustrated in C of FIG. 12, a person detected in an upperleft region as a person is a person and is far way, and thus athinning-out rate is set to a small value such as 1 or 2. Similarly, aperson detected in a lower right region is nearby, and thus athinning-out rate is set to 4 which is a larger value than that of theperson who is in a distant view.

In addition, a high rate of 10 is set as a thinning-out rate for aregion in which a person is not detected. For example, in a case inwhich thinning-out rates of 10 levels of 1 to 10 are set, it is possibleto prevent pixel values from being read out from the region when athinning-out rate is 10.

In a case in which it is intended to detect a person and perform anyprocessing on the person, a thinning-out rate is set to 10 for a portionother than the person so as to prevent the portion from being read out.In this manner, it is possible to drastically reduce the amount of datato be processed.

In this manner, it is also possible to detect a person and to set aresolution in accordance with the detected person. In addition, it isalso possible to set a resolution in accordance with a result of persondetection and a distance to the person.

Note that, in a case in which a resolution is set using not only aresult of person detection but also a distance, a configuration can beadopted in which the above-described processing is executed by addingthe distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.11.

Note that, here, although a case of detection of a person has beendescribed as an example, it is also possible to detect a portion of theperson, such as the face, hand, or foot of the person, and change aresolution of the portion. In addition, it is also possible to detect aportion of a person and change a resolution in accordance with adistance of the portion.

In addition, for example, a configuration may be adopted in which in acase in which a person's face is detected and the degree of matchingbetween the person's face and a face satisfying a predeterminedcondition, for example, a face registered in advance is high, aresolution of the region of the face is set to be high, and on thecontrary, when a face which is highly unlikely to be a face registeredin advance is detected, a resolution of the region of the face is set tobe high.

<Change of Resolution Based on Detection of Size>

Next, a case in which the size of a subject is detected and a resolutionis changed on the basis of the detected size will be described as anexample of a case in which a resolution is changed on the basis ofinformation other than a distance.

FIG. 13 is a diagram illustrating configuration examples of an imagingelement 221 and an image processing unit 231 when adopting aconfiguration in which a resolution is changed on the basis of detectionof the size. In a configuration of each of the imaging element 22F andthe image processing unit 23F illustrated in FIG. 13 and a configurationof each of the imaging element 22B and the image processing unit 23Billustrated in FIG. 3, the same components are denoted by the samereference numerals and signs, and a description thereof will beappropriately omitted.

The imaging element 22F illustrated in FIG. 13 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23F illustrated in FIG. 13 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a size detection unit 471.

The size detection unit 471 of the image processing unit 23F illustratedin FIG. 13 detects a subject imaged within a captured image processed bythe captured image processing unit 121 using the captured image, anddetects the size of the subject. A resolution map creating unit 108Fcreates a resolution map on the basis of the size of the subjectdetected by the size detection unit 471.

Processing performed by the imaging element 22F and the image processingunit 23F configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting the size of a subject.

A description of a case in which a resolution is changed by thedetection of a size will be added again with reference to FIG. 10. A ofFIG. 10 is a diagram illustrating an example of a captured image (animage to be processed). An input image 301 illustrated in A of FIG. 10is the same as the input image 301 illustrated in A of FIG. 7.

In a case in which such an input image 301 is acquired, a subject isdetected using the input image 301, and the size of the subject isdetected, an image 352 which is a result of size detection asillustrated in B of FIG. 10 is obtained. In B of FIG. 10, a regionsurrounded by a quadrangle is a region in which a subject (object) isdetected and is a region which is set as the size.

Any method may be used as a method of detecting a size. It is possibleto detect a size by detecting a subject within an input image 301 anddetecting the region of the subject as a size, and any method may beused for the detection of the subject in such a case.

In a case in which a size detection result image 352 as illustrated in Bof FIG. 10 is obtained as size detection information, a resolution isset for each region of the size detection result image 352. C of FIG. 10is a diagram illustrating an example in which a resolution is set foreach region and illustrating an example of a resolution map.

A resolution map 353 illustrated in C of FIG. 10 is the same as theresolution map 303 illustrated in C of FIG. 7, and indicates that thesmaller a numerical value within the resolution map 353 is, the lower athinning-out rate is, and the higher a numerical value is, the higher athinning-out rate is. In other words, in C of FIG. 10, the smaller anumerical value is, the higher a resolution is, and the higher anumerical value is, and the lower a resolution is.

It is possible to set a higher thinning-out rate (a lower resolution) asa detected size in a region in which a size is detected becomes larger.This is because it is considered that there is a strong likelihood that,in a case of a large size, a desired amount of information is obtainedeven when a resolution is lowered.

Basically, such setting is performed, but it is also possible to set aresolution in combination with a distance to a subject. C of FIG. 10illustrates the resolution map 353 in a case in which a resolution isset in consideration of such a distance.

For example, a resolution of a small subject in a distant view is set tobe high (maintained), and a resolution of a large subject in a near viewis set to be low. In the example illustrated in C of FIG. 10, as a size,a person detected in an upper left region has a small size and is faraway, and thus a thinning-out rate is set to a small value of 1 or 2. Inaddition, a person detected in a lower right region has a large size andis nearby, and thus a thinning-out rate is set to 4.

In this manner, in a case in which a resolution of a small subject in adistant view is set to be high and a resolution of a large subject in anear view is set to be low, it is possible to reduce the amount of datain a portion in a near view and to continuously detect a small subjectin a distant view without losing sight of the subj ect.

In addition, a resolution of a large subject in a distant view may beset to be low. Even in a distant view, the large subject is highlylikely to be detected even when a resolution is lowered, and thus it isalso possible to lower a resolution and to reduce the amount of data.

In addition, a resolution of a small subject in a near view may be setto be high. Even in a near view, the small subject is highly unlikely tobe detected when a resolution is lowered, and thus it is also possibleto set a resolution to a higher value and to detect such a small subjectwithout losing sight of the subject.

In this manner, a resolution is set in accordance with an object whichis a subject and a distance, and thus it is possible to realize theacquisition of efficient imaging data even when a large object is in thedistance and a small object is in a near view.

Such setting can be performed, but it is also possible to set aresolution in accordance with sizes of subjects positioned at the samedistance (a distance within a predetermined range). For example, aresolution of a large subject is set to be low and a resolution of asmall subject is set to be high among a plurality of subjects present atthe same distance.

In spite of the same distance, there is a strong likelihood that desiredinformation may be obtained from a large subject even when a resolutionis lowered, and thus a resolution is lowered. On the other hand, inspite of the same distance, there is a strong likelihood that desiredinformation may not be obtained from a small subject when a resolutionis lowered, and thus a resolution is maintained or increased withoutbeing lowered.

Also in a case in which a resolution is set in accordance with the sizeof a subject and a distance in this manner, it is possible to realizethe acquisition of efficient imaging data even when a large object ispresent in the distance and a small object is present in a near view andwhen both a large object and a small object are present in the samedistance.

A high rate of 10 may be set as a thinning-out rate for a region inwhich a size (subject) is not detected. For example, in a case in whichthinning-out rates of 10 levels of 1 to 10 are set, it is possible toprevent pixel values from being read out from the region when thethinning-out rate is 10. In this manner, it is possible to drasticallyreduce the amount of data to be processed.

In this manner, it is also possible to detect a size and to set aresolution in accordance with the detected size. In addition, it is alsopossible to set a resolution in accordance with a result of sizedetection and a distance to a subject of which the size is detected.

Note that, in a case in which a resolution is set using not only aresult of size detection but also a distance, a configuration can beadopted in which the above-described processing is executed by addingthe distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.13.

<Change of Resolution Based on Detection of Texture>

Next, a case in which texture or the amount of edge (hereinafter, adescription will be given using texture as an example) is detected and aresolution is changed on the basis of the detected texture will bedescribed as an example of a case in which a resolution is changed onthe basis of information other than a distance.

FIG. 14 is a diagram illustrating configuration examples of an imagingelement 22G and an image processing unit 23G when adopting aconfiguration in which a resolution is changed on the basis of detectionof the texture. In a configuration of each of the imaging element 22Gand the image processing unit 23G illustrated in FIG. 14 and aconfiguration of each of the imaging element 22B and the imageprocessing unit 23B illustrated in FIG. 3, the same components aredenoted by the same reference numerals and signs, and a descriptionthereof will be appropriately omitted.

The imaging element 22G illustrated in FIG. 14 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23G illustrated in FIG. 14 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a texture detection unit 491.

The texture detection unit 491 of the image processing unit 23Gillustrated in FIG. 14 detects texture using a captured image processedby the captured image processing unit 121. A resolution map creatingunit 108G creates a resolution map on the basis of the texture detectedby the texture detection unit 491.

Processing performed by the imaging element 22G and the image processingunit 23G configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting texture.

Any method may be used as a method of detecting texture. It is possibleto detect an edge (for example, a boundary between an object and abackground) within an input image 301 and to detect texture from theamount of edge. It is possible to detect, for example, a region otherthan the object by detecting the texture.

As texture, for example, it is considered that there is no much changein the amount of information in flat texture (texture of which the colordoes not much change, or the like) between a case in which a resolutionis set to be high and a case in which a resolution is set to be low.Accordingly, a low resolution (a high thinning-out rate) is set for aregion in which flat texture (a small amount of edge) is detected.

In addition, as texture, for example, a high resolution (a lowthinning-out rate) is set for a region in which texture (a large amountof edge) not being flat texture is detected, and is set to be aresolution with which texture can be reproduced as faithful as possible.

In this manner, it is possible to set a thinning-out rate in accordancewith the amount of edge.

In addition, it is also possible to set different resolution in a regionin which texture is detected and a region in which texture is notdetected. For example, a low resolution is set for the region in whichtexture is detected, and a high resolution is set for the region inwhich texture is not detected.

Basically, such setting is performed, but it is also possible to set aresolution in combination with a distance to a subject. For example, itis also possible to set a resolution in accordance with a distance to asubject and to set a resolution in accordance with the detection oftexture.

A resolution of a subject in a distant view is set to be high and aresolution of a subject in a near view is set to be low, for example,like the above-described change of a resolution based on a distance. Aresolution is set in this manner, and thus it is possible to detect asubject in a distant view without losing sight of the subject and toreduce the amount of data in a near view.

Further, even in a distant region, it is not necessary to acquire animage by increasing a resolution for a region determined to have textureor a small amount of edge, for example, a region such as sky, a roadsurface, or a flat wall, and thus a resolution is set to be low.

In this manner, it is possible to reduce the amount of data by loweringa resolution in accordance with texture or the amount of edge even in adistant region.

Note that, in a case in which a region of texture is detected and anyprocessing is performed on the detected texture, a resolution in theregion in which texture is detected can also be set to be higher thanresolutions in other regions in which texture is not detected.

In this manner, it is also possible to detect texture and to set aresolution in accordance with the detected texture. In addition, aresolution can also be set in accordance with a result of texturedetection and a distance to the texture.

Note that, in a case in which a resolution is set using not only aresult of texture detection but also a distance, a configuration can beadopted in which the above-described processing is executed by addingthe distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.14.

<Change of Resolution Based on Detection of Type>

Next, a case in which a type of subject is detected and a resolution ischanged on the basis of the detected type will be described as anexample of a case in which a resolution is changed on the basis ofinformation other than a distance. The type of subject is a material ofan object such as cloth or a metal. In addition, the type of subjectalso includes types such as a person and a car.

FIG. 15 is a diagram illustrating configuration examples of an imagingelement 22H and an image processing unit 23H when adopting aconfiguration in which a resolution is changed on the basis of detectionof a type. In a configuration of each of the imaging element 22H and theimage processing unit 23H illustrated in FIG. 15 and a configuration ofeach of the imaging element 22B and the image processing unit 23Billustrated in FIG. 3, the same components are denoted by the samereference numerals and signs, and a description thereof will beappropriately omitted.

The imaging element 22H illustrated in FIG. 15 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23H illustrated in FIG. 15 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a type detection unit 511.

The type detection unit 511 of the image processing unit 23H illustratedin FIG. 15 detects a subject using a captured image processed by thecaptured image processing unit 121 and detects the type of subject. Aresolution map creating unit 108H creates a resolution map on the basisof a type detected by the type detection unit 511.

Processing performed by the imaging element 22H and the image processingunit 23H configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting the type of a subject.

As a method of detecting a type, any method may be used, or a generalrecognition technique or the like can be used. For example, a persondetection technique, a car detection technique, or the like can be used,and a method based on a technique such as machine learning or DeepLearning can be used. In addition, gloss, a pattern, and the like aredifferent depending on a material, and such features are extracted, sothat a type may be detected.

In a case in which a material such as a metal or wood, a face, a licenseplate, or the like is detected as a type and such a type is detected, aresolution can be set in accordance with the detected type. For example,in a case in which a license plate or the like is detected, a resolutionis set to be high so that characters can be read clearly.

Basically, setting is performed in accordance with such a type, but itis also possible to set a resolution in combination with a distance to asubject. For example, it is also possible to set a resolution inaccordance with a distance to a subject and to set a resolution inaccordance with detection of a type.

A resolution of a subject in a distant view is set to be high and aresolution of a subject in a near view is set to be low, for example,like the above-described change of a resolution based on a distance. Aresolution is set in this manner, and thus it is possible to detect asubject in a distant view without losing sight of the subject and toreduce the amount of data in a near view.

Further, even in a near view, a resolution is set to be high, forexample, for a region classified as a type such as a license plate or aface in accordance with a type.

In this manner, a resolution is increased in accordance with a type evenin a near view, and thus it is possible to further improve the accuracyof detection of a predetermined object.

In this manner, usually, the detection of a small object in the distanceis also robust, and it is possible to set a resolution to be high so asto further obtain detailed data for a region having a type of objectdesired to be noticed while reducing the amount of data by thinning outa near view, whereby it is possible to realize the acquisition ofefficient imaging data.

Note that, in a case in which a resolution is set using not only aresult of type detection but also a distance, a configuration can beadopted in which the above-described processing is executed by addingthe distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.15.

<Change of Resolution Based on Detection of Amount of Movement>

Next, a case in which the amount of movement of a subject is detectedand a resolution is changed on the basis of the detected amount ofmovement will be described as an example of a case in which a resolutionis changed on the basis of information other than a distance.

FIG. 16 is a diagram illustrating configuration examples of an imagingelement 221 and an image processing unit 231 when adopting aconfiguration in which a resolution is changed on the basis of detectionof the amount of movement. In a configuration of each of the imagingelement 221 and the image processing unit 231 illustrated in FIG. 16 anda configuration of each of the imaging element 22B and the imageprocessing unit 23B illustrated in FIG. 3, the same components aredenoted by the same reference numerals and signs, and a descriptionthereof will be appropriately omitted.

The imaging element 221 illustrated in FIG. 16 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 231 illustrated in FIG. 16 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a movement amount detection unit 531.

The movement amount detection unit 531 of the image processing unit 231illustrated in FIG. 16 detects a subject using a captured imageprocessed by the captured image processing unit 121 and detects theamount of movement of the subject. A resolution map creating unit 1081creates a resolution map on the basis of the amount of movement detectedby the movement amount detection unit 531.

Processing performed by the imaging element 221 and the image processingunit 231 configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting the amount of movement of a subject.

Any method may be used as a method of detecting the amount of movement.For example, a moving body may be detected by comparing pixel valueswith each other using the input image 301 for two frames, and the amountof movement of the moving body may further be detected.

A resolution in a region having a subject (moving body) is set to be lowin a case in which the amount of movement of the subject is large, and aresolution in a region having a subject (moving body) is set to be highin a case in which the amount of movement of the subject is small. Forexample, in a case in which a region of a moving body having a smallamount of movement and moving across only several pixels in units ofpixels is thinned out, there is a likelihood that an object cannot bedetected with a high level of accuracy (the amount of movement of theobj ect cannot be detected with a high level of accuracy), and thus itis possible to set a resolution to be high for an object having a smallamount of movement.

Basically, setting is performed in accordance with such an amount ofmovement, but it is also possible to set a resolution in combinationwith a distance to a subject. For example, it is also possible to set aresolution in accordance with a distance to a subject and to set aresolution in accordance with detection of an amount of movement.

A resolution of a subject in a distant view is set to be high and aresolution of a subject in a near view is set to be low, for example,like the above-described change of a resolution based on a distance. Aresolution is set in this manner, and thus it is possible to detect asubject in a distant view without losing sight of the subject and toreduce the amount of data in a near view.

Further, even in a near view, a resolution is set to be high, forexample, for a region having an object moving with a different amount ofmovement from that of a surrounding object in accordance with the amountof movement. For example, a speeding car is traveling at a speeddifferent from those of surrounding cars traveling at speeds which arenot speeding. It is possible to detect such a car traveling at a speeddifferent from those of surrounding cars and to set a high resolution ina region in which the car is imaged so as to follow the car.

In addition, for example, in a case in which there is a person who ismoving at a speed different from those of other persons in a crowd, inother words, there is a person who is running away, it is possible todetect such a person who is moving at a speed different from those ofthe surrounding persons and to set a high resolution in a region inwhich the person is imaged so as to follow the person.

In this manner, it is possible to further improve the accuracy ofdetection of a predetermined object by increasing a resolution even in anear view, in accordance with the amount of movement.

In this manner, usually, the detection of a small object in the distanceis also robust, and it is possible to set a resolution to be high so asto further obtain detailed data for the region in a case in which anabnormality occurs while reducing the amount of data by thinning out anear view, and to realize the acquisition of efficient imaging data.

Note that, in a case in which a resolution is set using not only aresult of movement amount detection but also a distance, a configurationcan be adopted in which the above-described processing is executed byadding the distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.16.

<Change of Resolution Based on Detection of Moving Direction>

Next, a case in which a moving direction of a subject is detected and aresolution is changed on the basis of the detected moving direction willbe described as an example of a case in which a resolution is changed onthe basis of information other than a distance.

FIG. 17 is a diagram illustrating configuration examples of an imagingelement 22J and an image processing unit 23J when adopting aconfiguration in which a resolution is changed on the basis of detectionof the moving direction. In a configuration of each of the imagingelement 22J and the image processing unit 23J illustrated in FIG. 17 anda configuration of each of the imaging element 22B and the imageprocessing unit 23B illustrated in FIG. 3, the same components aredenoted by the same reference numerals and signs, and a descriptionthereof will be appropriately omitted.

The imaging element 22J illustrated in FIG. 17 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23J illustrated in FIG. 17 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a moving direction detection unit 551.

The moving direction detection unit 551 of the image processing unit 23Jillustrated in FIG. 17 detects a subject using a captured imageprocessed by the captured image processing unit 121 and detects themoving direction of the subject. A resolution map creating unit 108Jcreates a resolution map on the basis of the moving direction detectedby the moving direction detection unit 551.

Processing performed by the imaging element 22J and the image processingunit 23J configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting the moving direction of a subject.

Any method may be used as a method of detecting the moving direction.For example, a moving body may be detected by comparing pixel valueswith each other using the input image 301 for two frames, and the movingdirection of the moving body may further be detected.

In a case in which there is a subject having a moving directiondifferent from those of other subject, a resolution in a region havingthe subject (moving body) is set to be high. For example, in a case inwhich the imaging device 10 (FIG. 1) is used for the purpose of asurveillance camera and a predetermined place, for example, a ticketgate or a one-way road is imaged, a person or a car basically moves inthe same direction in such a place.

There is a strong likelihood that a person or a car moving in adifferent direction is a suspicious object in such a place. Accordingly,in a case in which such an object moving in a direction different fromthose of other objects is detected, a resolution in a region of theobject is set to be high.

Basically, setting is performed in accordance with such a movingdirection, but it is also possible to set a resolution in combinationwith a distance to a subject. For example, it is also possible to set aresolution in accordance with a distance to a subject and to set aresolution in accordance with detection of a moving direction.

A resolution of a subject in a distant view is set to be high and aresolution of a subject in a near view is set to be low, for example,like the above-described change of a resolution based on a distance. Aresolution is set in this manner, and thus it is possible to detect asubject in a distant view without losing sight of the subject and toreduce the amount of data in a near view.

Further, even in a near view, a resolution is set to be high, forexample, for a region having an object moving in a moving directiondifferent from those of the surrounding objects, in accordance with amoving direction.

In this manner, it is possible to further improve the accuracy ofdetection of a predetermined object by increasing a resolution even in anear view, in accordance with the moving direction.

In this manner, usually, the detection of a small object in the distanceis also robust, and it is possible to set a resolution to be high so asto further obtain detailed data for the region in a case in which anabnormality occurs while reducing the amount of data by thinning out anear view, and to realize the acquisition of efficient imaging data.

Note that, in a case in which a resolution is set using not only aresult of moving direction detection but also a distance, aconfiguration can be adopted in which the above-described processing isexecuted by adding the distance information acquisition unit 131 and thedistance information generation unit 109 to the configurationillustrated in FIG. 17.

<Change of Resolution Based on Detection of Load>

Next, a case in which a load of a network or a processing unit isdetected and a resolution is changed on the basis of the detected loadwill be described as an example of a case in which a resolution ischanged on the basis of information other than a distance.

FIG. 18 is a diagram illustrating configuration examples of an imagingelement 22K and an image processing unit 23K when adopting aconfiguration in which a resolution is changed on the basis of detectionof a load. In a configuration of each of the imaging element 22K and theimage processing unit 23H illustrated in FIG. 18 and a configuration ofeach of the imaging element 22B and the image processing unit 23Billustrated in FIG. 3, the same components are denoted by the samereference numerals and signs, and a description thereof will beappropriately omitted.

The imaging element 22K illustrated in FIG. 18 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23K illustrated in FIG. 18 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a load detection unit 571.

The load detection unit 571 of the image processing unit 23K illustratedin FIG. 18 detects a load at a predetermined point in time from loadinformation supplied from the outside of the image processing unit 23K.A resolution map creating unit 108K creates a resolution map on thebasis of the load detected by the load detection unit 571.

Processing performed by the imaging element 22K and the image processingunit 23K configured in such a manner is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, in step S101, distance information isgenerated, but there is a difference in that this process is replacedwith a process of detecting a load.

For example, the load detection unit 571 acquires a congestion state ofa network as load information through the communication unit 30 (FIG.1), and a resolution of a captured image is set to be low in order toreduce the amount of imaging data input through the network in a case inwhich it is determined that the network is congested.

In addition, for example, the load detection unit 571 acquires aprocessing load of the captured image processing unit 121 as loadinformation, and a resolution of a captured image to be imaged andprocessed by the imaging element 22K is set to be low in order to reducethe amount of imaging data to be input to the captured image processingunit 121, in other words, imaging data to be supplied from the imagingelement 22K in a case in which it is determined that a processing loadof the captured image processing unit 121 is increasing.

A processing load within the imaging device 10 is acquired, and themeasured information may be acquired as load information.

Note that, here, a description has been given on the assumption that aresolution is lowered when a processing load is increased, but a processof increasing a resolution may be performed when there is a margin inthe processing load.

Basically, setting is performed in accordance with such a load, but itis also possible to set a resolution in combination with a distance to asubject. For example, it is also possible to set a resolution inaccordance with a distance to a subject and to set a resolution inaccordance with detection of a load.

When there is a margin in a case in which a processing load within theimaging device 10 is low, a case in which a network is not congested, orthe like, it is not necessary to lower a resolution, and thus aresolution is set to be a high resolution (setting for lowering athinning-out rate and not performing thinning-out is performed), so thata high-quality image is output in all regions within the image.

When a load within the imaging device 10 or a load of a network is highand it is necessary to lower a resolution, the amount of data of animage is reduced by setting a high resolution for a subject in a distantview and setting a low resolution for a subject in a near view, forexample, like the above-described change of a resolution based on adistance.

In this manner, it is possible to prevent unnecessary degradation of aresolution and deterioration of image quality even when there is amargin in resources.

In this manner, it is possible to keep detection robust by changing aresolution in accordance with a distance without uniformly lowering aresolution within an image when a load is high.

Note that, in a case in which a resolution is set using not only aresult of load detection but also a distance, a configuration can beadopted in which the above-described processing is executed by addingthe distance information acquisition unit 131 and the distanceinformation generation unit 109 to the configuration illustrated in FIG.18.

<Change of Resolution Based on Determination of Resolution>

Next, a case in which a resolution is determined by an algorithm to beapplied to a latter stage and a resolution is changed on the basis ofthe determined resolution will be described as an example of a case inwhich a resolution is changed on the basis of information other than adistance.

FIG. 19 is a diagram illustrating configuration examples of an imagingelement 22L and an image processing unit 23L when adopting aconfiguration in which a resolution is changed on the basis of analgorithm to be applied to a latter stage. In a configuration of each ofthe imaging element 22L and the image processing unit 23L illustrated inFIG. 19 and a configuration of each of the imaging element 22B and theimage processing unit 23B illustrated in FIG. 3, the same components aredenoted by the same reference numerals and signs, and a descriptionthereof will be appropriately omitted.

The imaging element 22L illustrated in FIG. 19 has the sameconfiguration as that of the imaging element 22B illustrated in FIG. 3.The image processing unit 23L illustrated in FIG. 19 is different fromthe image processing unit 23B illustrated in FIG. 3 in that the distanceinformation generation unit 109 of the image processing unit 23B isreplaced with a resolution determination unit 591.

The resolution determination unit 591 of the image processing unit 23Lillustrated in FIG. 19 determines a resolution from resolutiondetermination information supplied from the outside of the imageprocessing unit 23L. A resolution map creating unit 108L creates aresolution map on the basis of the resolution determined by theresolution determination unit 591.

A process performed by the imaging element 22L and the image processingunit 23L having such a configuration is basically performed on the basisof the flowchart illustrated in FIG. 6, and thus a description thereofwill be omitted. However, although distance information is generated instep S101, but there is a difference in that this process is replacedwith a process of determining a resolution.

For example, the resolution determination unit 591 determines aresolution satisfying a resolution required by an algorithm in a latterstage. The algorithm in the latter stage is, for example, facerecognition, moving body detection, or the like.

In a case in which the algorithm in the latter stage is facerecognition, the determination of a resolution for setting a highresolution for a region detected as a face and setting a low resolutionfor other regions is performed. In addition, for example, in a case inwhich the algorithm in the latter stage is detection of a moving body,the determination of a resolution for setting a high resolution for aregion detected as a moving body and setting a low resolution for otherregions is performed.

The resolution determination unit 591 acquires resolution determinationinformation indicating to what extent the algorithm in the latter stagerequires a resolution from the outside to determine a resolution. Inaddition, the resolution determination unit 591 stores a table in whichan algorithm and a resolution are associated with each other, and maydetermine a resolution with reference to the table.

Basically, setting is performed in accordance with such a determinationof a resolution, but it is also possible to set a resolution incombination with a distance to a subject. For example, it is alsopossible to set a resolution in accordance with a distance to a subjectand to set a resolution in accordance with detection of a determinationof a resolution.

A resolution of a subject in a distant view is set to be high and aresolution of a subject in a near view is set to be low, for example,like the above-described change of a resolution based on a distance. Aresolution is set in this manner, and thus it is possible to detect asubject in a distant view without losing sight of the subject and toreduce the amount of data in a near view.

Further, even in a near view such as a case in which the algorithm inthe latter stage is an algorithm using an edge as important informationsuch as segmentation (region division), a resolution in a region in thevicinity of the edge may be set to be high.

In addition, even in a near view such as a case in which the algorithmin the latter stage is an algorithm using the texture of a subject asimportant information, a resolution in a region of the subject may beset to be high. In addition, it is also possible to lower a resolutionas necessary such as a case in which the algorithm in the latter stageis an algorithm requiring imaging data less influenced by noise.

Note that an example is described here. The present technology can beapplied in a case in which a resolution is changed within a range inwhich a resolution required by an algorithm (application) is maintained.

A resolution is changed in this manner, so that the detection of a smallobject in the distance is also robust and it is possible to generate andsupply data appropriate for the algorithm in the latter stage(application) while reducing the amount of data by thinning out a nearview, whereby it is possible to realize the acquisition of efficientimaging data.

Note that, in a case in which a resolution is set using not only aresult of resolution determination detection but also a distance, aconfiguration can be adopted in which the above-described processing isexecuted by adding the distance information acquisition unit 131 and thedistance information generation unit 109 to the configurationillustrated in FIG. 19.

<Change of Resolution Based on Lens>

Next, a case in which a resolution is changed on the basis of the typeof lens group 21 (FIG. 1) of the imaging device 10 will be described asan example of a case in which a resolution is changed on the basis ofinformation other than a distance.

For example, as read-out in a case in which a lens having a shallowdepth of field is mounted, a high resolution is set for a region at adistance in the vicinity of a main subject, and a low resolution is setfor regions (regions in which a blur occurs in the lens) at the otherdistances.

A resolution is set in this manner, and thus it is possible toefficiently read out the other original regions in which a blur occursin the lens with a low resolution, while obtaining a resolution of amain subject.

<Configuration in which Two or more Images are Output>

Incidentally, for example, according to the configuration of the imageprocessing unit 23D illustrated in FIG. 9, image data of an image havingan adjusted resolution is supplied from the imaging element 22D to theimage processing unit 23D. The moving body detection unit 431 detects amoving body using an image supplied from the captured image processingunit 121, in other words, an image supplied to the image processing unit23D.

With such a configuration, the moving body detection unit 431 detects amoving body using an image having an adjusted resolution, and thus themoving body detection unit 431 processes an image having an unfixedresolution such as when a moving body is detected using an image havinga high resolution or when a moving body is detected using an imagehaving a low resolution. Thus, there is a likelihood that the accuracyof detection of a moving body may not be stabilized due to the movingbody detection unit 431.

Consequently, configurations of an imaging element 22M and an imageprocessing unit 23M may be adopted as a configuration as illustrated inFIG. 20. The imaging element 22M illustrated in FIG. 20 basically hasthe same configuration as that of the imaging element 22B illustrated inFIG. 3, but there is a difference in that an output control unit 106M isconfigured to output N streams. For example, the output control unit106M is configured to be capable of outputting streams of image datahaving different resolutions in parallel.

The output control unit 106M outputs an image having a predeterminedresolution, for example, resolution such as VGA to a moving bodydetection unit 431M of the image processing unit 23M, as streams ofimage data having different resolutions. An image to be supplied to themoving body detection unit 431M is an image which is not controlled bythe resolution control unit 107 and has a fixed resolution at all times.

In addition, the output control unit 106M outputs an image having achanged resolution to a captured image processing unit 121 of the imageprocessing unit 23M as streams of image data having differentresolutions by receiving an instruction from the resolution control unit107.

In a case in which the imaging element 22M is configured in such amanner, the moving body detection unit 431M receives the supply of animage having a fixed resolution from the output control unit 106M of theimaging element 22M and detects a moving body using the image.Accordingly, it is possible to stably detect the moving body.

In addition, an image having a changed resolution is supplied to thecaptured image processing unit 121 in accordance with an instructiongiven from the resolution control unit 107, and thus it is possible toreceive image data of which the amount of data is appropriately reduced.

In this manner, a plurality of streams are output from the imagingelement 22M. One of the streams can be set to be a stream of an image ofwhich the resolution has not been changed, and another stream can be setto be a stream of an image of which the resolution has been changed.

In addition, an image of which the resolution has not been changed isprovided, and thus it is possible to perform stable processing, forexample, the detection of a moving body.

Note that, here, although the imaging device 10 performing detection ofa moving body illustrated in FIG. 9 has been described as an example, astructure in which a plurality of streams are output and processed canbe applied to the imaging device 10 illustrated in FIGS. 10 to 19.Further, for example, in the image processing unit 23B within) theimaging device 10 illustrated in FIG. 3, a structure in which aplurality of streams are output and processed can also be applied to acase in which distance information is generated using an image obtainedfrom the imaging element 22B.

<Usage Example>

FIG. 21 is a diagram illustrating a usage example of the above-describedimaging device 1.

The imaging device can be, for example, used in various cases in whichlight such as visible light, infrared light, ultraviolet light and X-rayis sensed as described below.

-   Devices that take images used for viewing, such as a digital camera    and a portable appliance with a camera function.-   Devices used for traffic, such as an in-vehicle sensor that takes    images of the front and the back of a car, surroundings, the inside    of the car, and the like, a monitoring camera that monitors    travelling vehicles and roads, and a distance sensor that measures    distances between vehicles and the like, which are used for safe    driving (e.g., automatic stop), recognition of the condition of a    driver, and the like.-   Devices used for home electrical appliances, such as a TV, a    refrigerator, and an air conditioner, to takes images of a gesture    of a user and perform appliance operation in accordance with the    gesture.-   Devices used for medical care and health care, such as an endoscope    and a device that performs angiography by reception of infrared    light.-   Devices used for security, such as a monitoring camera for crime    prevention and a camera for personal authentication.-   Devices used for beauty care, such as skin measurement equipment    that takes images of the skin and a microscope that takes images of    the scalp.-   Devices used for sports, such as an action camera and a wearable    camera for sports and the like.-   Devices used for agriculture, such as a camera for monitoring the    condition of the field.

<Recording Medium>

A series of processes described above can be executed by hardware orsoftware. When a series of processes is executed by software, a programconstituting the software is installed in a computer. Here, examples ofthe computer include a computer incorporated in dedicated hardware and ageneral-purpose personal computer which is capable of executing variouskinds of functions when various kinds of programs are installed therein.

FIG. 22 is a block diagram illustrating an exemplary hardwareconfiguration of a computer that executes a series of processingdescribed above by a program. In a computer, a central processing unit(CPU) 1001, a read only memory (ROM) 1002, and a random access memory(RAM) 1003 are connected to one another by a bus 1004. An input/outputinterface 1005 is further connected to the bus 1004. An input unit 1006,an output unit 1007, a storage unit 1008, a communication unit 1009, anda drive 1010 are connected to the input/output interface 1005.

The input unit 1006 includes a keyboard, a mouse, a microphone, or thelike. The output unit 1007 includes a display, a speaker, or the like.The storage unit 1008 includes a hard disk, a nonvolatile memory, or thelike. The communication unit 1009 includes a network interface or thelike. The drive 1010 drives a removable medium 1011 such as a magneticdisk, an optical disk, a magneto-optical disk, or a semiconductormemory.

In the computer configured as described above, the CPU 1001 loads aprogram that is stored, for example, in the storage unit 1008 onto theRAM 1003 via the input/output interface 1005 and the bus 1004, andexecutes the program. Thus, the above-described series of processing isperformed.

Programs to be executed by the computer (the CPU 1001) are providedbeing recorded in the removable medium 1011 which is a packaged mediumor the like. Also, programs may be provided via a wired or wirelesstransmission medium, such as a local area network, the Internet ordigital satellite broadcasting.

In the computer, by inserting the removable medium 1011 into the drive1010, the program can be installed in the storage unit 1008 via theinput/output interface 1005. Further, the program can be received by thecommunication unit 1009 via a wired or wireless transmission medium andinstalled in the storage unit 1008. Moreover, the program can beinstalled in advance in the ROM 1002 or the storage unit 1008.

It should be noted that the program executed by the computer may be aprogram that is processed in time series according to the sequencedescribed in this specification or a program that is processed inparallel or at necessary timing such as upon calling.

Also, in this specification, the term “system” represents the totalityof an apparatus composed of a plurality of apparatus.

Note that the effects described in the present specification are notlimiting but are merely examples, and there may be additional effects.

Note that an embodiment of the disclosure is not limited to theembodiments described above, and various changes and modifications maybe made without departing from the scope of the disclosure.

Additionally, the present technology may also be configured as below.

-   (1)

An imaging device including:

a control unit which changes a resolution of a captured image on thebasis of distance information, corresponding to the captured image,regarding a detected distance to a subject included in the image.

-   (2)

The imaging device according to (1),

in which the control unit changes a resolution of a portion of a regionof the captured image on the basis of the distance information.

(3)

The imaging device according to (2),

in which the portion of the region is a region distant from anotherregion, and

the control unit changes the resolution of the portion of the regionsuch that the portion of the region becomes higher than a resolution ofanother region.

-   (4)

The imaging device according to any of (1) to (3),

in which the control unit sets a resolution of the subject to be high ina case in which the distance to the subject is longer than apredetermined reference, and sets the resolution of the subject to below in a case in which the distance to the subject is smaller than thepredetermined reference.

-   (5)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis of thedistance information of a moving body detected in the captured image.

-   (6)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis of thedistance information of a person detected in the captured image.

-   (7)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis of a sizeof the subject detected in the captured image and the distanceinformation.

-   (8)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis oftexture detected in the captured image and the distance information.

-   (9)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis of a typeof the subject detected in the captured image and the distanceinformation.

-   (10)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis of anamount of movement of the subject detected in the captured image and thedistance information.

-   (11)

The imaging device according to any of (1) to (4),

in which the control unit controls the resolution on the basis of amoving direction of the subject detected in the captured image and thedistance information.

-   (12)

The imaging device according to any of (1) to (4),

in which the control unit controls a resolution in accordance with thedistance information and a processing load.

-   (13)

The imaging device according to any of (1) to (4),

in which the control unit controls a resolution in accordance with thedistance information and an algorithm for performing a process using theimage.

-   (14)

The imaging device according to any of (1) to (4),

in which the control unit controls a resolution in accordance with thedistance information and a mounted lens.

-   (15)

The imaging device according to any of (1) to (14),

in which an image of which the resolution is controlled by the controlunit and an image having a fixed resolution are output.

-   (16)

The imaging device according to any of (1) to (15),

in which the control unit changes a resolution by changing athinning-out rate of pixel read-out.

-   (17)

The imaging device according to (16),

in which a process of lowering a resolution by thinning out pixelswithin a predetermined region is performed by adding pixel values ofpixels having a same color and dividing a value obtained by the additionby the number of pixels in the addition.

-   (18)

The imaging device according to any of (1) to (17), further including:

an imaging unit which captures the image.

-   (19)

An imaging method including:

a step of changing a resolution of a captured image on the basis ofdistance information, corresponding to the captured image, regarding adetected distance to a subject included in the image.

-   (20)

A program causing a computer to execute a process including:

a step of changing a resolution of a captured image on the basis ofdistance information, corresponding to the captured image, regarding adetected distance to a subject included in the image.

REFERENCE SIGNS LIST

-   10 imaging device-   21 lens group-   22 imaging element-   23 image processing unit-   24 frame memory-   25 display unit-   26 recording unit-   27 operation unit-   28 power supply-   29 driving unit-   30 communication unit-   31 bus line-   101 pixel array portion-   102 read-out control unit-   103 memory interface-   104 memory-   105 signal processing unit-   106 output control unit-   107 resolution control unit-   108 resolution map creating unit-   109 distance information generation unit-   121 captured image processing unit-   131 distance information acquisition unit-   431 moving body detection unit-   451 person detection unit-   471 size detection unit-   491 texture detection unit-   511 type detection unit-   531 movement amount detection unit-   551 moving direction determination unit-   571 load detection unit-   591 resolution determination unit

1. An imaging device comprising: a control unit which changes aresolution of a captured image on a basis of distance information,corresponding to the captured image, regarding a detected distance to asubject included in the image.
 2. The imaging device according to claim1, wherein the control unit changes a resolution of a portion of aregion of the captured image on a basis of the distance information. 3.The imaging device according to claim 2, wherein the portion of theregion is a region distant from another region, and the control unitchanges the resolution of the portion of the region such that theportion of the region becomes higher than a resolution of anotherregion.
 4. The imaging device according to claim 1, wherein the controlunit sets a resolution of the subject to be high in a case in which thedistance to the subject is longer than a predetermined reference, andsets the resolution of the subject to be low in a case in which thedistance to the subject is smaller than the predetermined reference. 5.The imaging device according to claim 1, wherein the control unitcontrols the resolution on a basis of the distance information of amoving body detected in the captured image.
 6. The imaging deviceaccording to claim 1, wherein the control unit controls the resolutionon a basis of the distance information of a person detected in thecaptured image.
 7. The imaging device according to claim 1, wherein thecontrol unit controls the resolution on a basis of a size of the subjectdetected in the captured image and the distance information.
 8. Theimaging device according to claim 1, wherein the control unit controlsthe resolution on a basis of texture detected in the captured image andthe distance information.
 9. The imaging device according to claim 1,wherein the control unit controls the resolution on a basis of a type ofthe subject detected in the captured image and the distance information.10. The imaging device according to claim 1, wherein the control unitcontrols the resolution on a basis of an amount of movement of thesubject detected in the captured image and the distance information. 11.The imaging device according to claim 1, wherein the control unitcontrols the resolution on a basis of a moving direction of the subjectdetected in the captured image and the distance information.
 12. Theimaging device according to claim 1, wherein the control unit controls aresolution in accordance with the distance information and a processingload.
 13. The imaging device according to claim 1, wherein the controlunit controls a resolution in accordance with the distance informationand an algorithm for performing a process using the image.
 14. Theimaging device according to claim 1, wherein the control unit controls aresolution in accordance with the distance information and a mountedlens.
 15. The imaging device according to claim 1, wherein an image ofwhich the resolution is controlled by the control unit and an imagehaving a fixed resolution are output.
 16. The imaging device accordingto claim 1, wherein the control unit changes a resolution by changing athinning-out rate of pixel read-out.
 17. The imaging device according toclaim 16, wherein a process of lowering a resolution by thinning outpixels within a predetermined region is performed by adding pixel valuesof pixels having a same color and dividing a value obtained by theaddition by a number of pixels in the addition.
 18. The imaging deviceaccording to claim 1, further comprising: an imaging unit which capturesthe image.
 19. An imaging method comprising: a step of changing aresolution of a captured image on a basis of distance information,corresponding to the captured image, regarding a detected distance to asubject included in the image.
 20. A program causing a computer toexecute a process comprising: a step of changing a resolution of acaptured image on a basis of distance information, corresponding to thecaptured image, regarding a detected distance to a subject included inthe image.